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ABSTRACT 
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development; (3) teaching /I earning; (4) course structure; (5) student 
support; (6) faculty support; and (7) evaluation and assessment. The study 
seeks to ascertain the degree to which the benchmarks are actually 
incorporated in the policies and practices of the institutions, and how 
important the benchmarks are to faculty, administrators, and students. 
Quantitative information for the study was derived from a Likert- scale 
survey, and qualitative information was derived from in-depth interviews. The 
report concludes that, for the most part, the benchmarks are considered 
important and that the institutions strive to incorporate them into their 
policies, practices, and procedures. Appendixes contain institutional 
profiles and detailed survey results. (Contains 26 references.) (RH) 
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Foreword 



T he public debate over the merits of Internet- 
based distance learning too often consists of 
high-pitched vitriol and hyperbole. Proponents 
ooze with blind adoration, declaring that online 
learning can resolve all the problems confronting 
traditional education. Opponents insist that courses 
taught on the net are incapable of living up to 
the standards of the traditional bricks and 
mortar classroom. 

While many continue to divide themselves into these 
"for" and "against" camps, a first-of-its-kind study 
produced by The Institute for Higher Education Policy 
and sponsored by the National Education 
Association, the nations' largest professional 
association of higher education faculty, and 
Blackboard Inc., a leading Internet education 
company, brings reason and research data to this 
overheated debate, providing more tangible measures 
of quality in distance learning. The study makes clear 
that distance learning can be quality learning. 

Quality On the Line identifies 24 benchmarks 
considered essential to ensuring excellence in Internet- 
based distance learning. The benchmarks are divided 
into seven categories of quality measures currently in 
use on campuses around the nation. 

These benchmarks distill the best strategies used by 
colleges and universities that are actively engaged in 
online learning, ensuring quality for the students and 
faculty who use it. This area of study is particularly 



important considering the breakneck pace of growth 
in Internet-based distance learning. 

This growth is no surprise. The opportunities for learning 
and growth online are virtually limitless. Internet-based 
education transcends typical time and space barriers, 
giving students the ability to access learning opportunities 
day and night from every comer of the globe. Coursework 
can now provide material in highly interactive audio, 
video, and textual formats at a pace set by the student. 

These attributes are very exciting, but drastically 
different from the traditional classroom-based 
education. How can a teaching and learning process 
that deviates so markedly from what has been practiced 
for hundreds of years maintain quality? Who will be 
the guardians of quality and the innovators to present 
material to students in new and engaging ways? 

As the nation's largest education union and as a leading 
online education company, we believe the distance from 
student to teacher must be measured in results — quality 
learning — achieved by our students. The benchmarks 
identified in this study will ensure that as we plunge 
headlong into a technology-driven future, unions and 
business help America's campuses remain on the forward 
edge of quality. 

Matthew Pittinsky, 

Chairman 
Blackboard Inc. 



Bob Chase, President 
National Education 
Association 
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Executive Summary 



I n the 10 years since the coding language for the 
World Wide Web (WWW) was developed, 
educational institutions, research centers, libraries, 
government agencies, commercial enterprises, 
advocacy groups, and a multitude of individuals have 
rushed to connect to the Internet. One of the 
consequences of this tremendous surge in online 
communication has been the rapid growth of 
technology-mediated distance learning at the higher 
education level. A recent survey by the U.S. 
Department of Education's National Center for 
Education Statistics (NCES) found that from 1994- 
95 to 1997-98 the number of distance education 
programs increased by 72 percent. Moreover, an 
additional 20 percent of the institutions surveyed plan 
to establish distance education programs within the 
next three years. The survey estimated that more than 
1.6 million students were enrolled in distance 
education courses in 1997-98.^ 

This extraordinary growth of technology-mediated 
distance learning in higher education has prompted 
several different organizations to develop principles, 
guidelines, or benchmarks to ensure quality distance 
education. The quality assurance benchmarks 
promoted by these organizations are designed to apply 
to a wide variety of institutional contexts and consist 
of fairly broad statements. Virtually all of the strategies 
include such topics as course development, faculty 
training, student services, learning resources, 
infrastructure, and outcomes assessment. 

These benchmarks, which were developed initially for 
all types of distance learning, have existed in various 
forms for a number of years. The question that arises 
is whether they are applicable to Internet-based 
distance education. In short, are the current 
benchmarks appropriate and necessary to ensure 



quality Internet-based distance education? Two 
organizations — the National Education Association 
(NEA), the nation's largest professional association 
of higher education faculty, and Blackboard Inc., a 
widely used platform provider for online education — 
are interested in exploring these issues and their 
implications. The two organizations jointly 
commissioned The Institute for Higher Education 
Policy to examine the benchmarks by studying active 
distance learning programs at several institutions. 

The Institute was approached by the two 
commissioning organizations in part because of its 
previous experience in analyzing issues related to 
quality in distance education. The Institute's widely 
cited 1999 report. What's the Difference? A Review of 
Contemporary Research on the Effectiveness of Distance 
Learning in Higher Education, has generated 
considerable dialogue throughout academia about 
what constitutes quality in distance learning settings. 
This report is not intended to overcome many of the 
limitations of previous research noted in What's the 
Difference? Instead, it uses case studies to help build a 
foundation for future analyses capable of refining or 
expanding upon the lessons learned from the 
institutions studied here. 

Specifically, NEA and Blackboard Inc. asked The 
Institute to attempt to validate those benchmarks that 
have been published by various entities, with specific 
attention to Internet-based distance education. This 
study is designed to ascertain the degree to which the 
benchmarks are actually incorporated in the policies, 
procedures, and practices of colleges and universities 
that are distance education leaders. In addition, this 
case study seeks to determine how important the 
benchmarks are to the institutions' faculty, 
administrators, and students. 



This count is not unduplicated; the survey tallied students in each distance learning course in which they enrolled, not as individual 
students who may have enrolled in one or more distance learning courses. 
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The case study process consisted of three sequential 
phases. First, a comprehensive literature search was 
conduaed to compile those benchmarks recommended 
by other organizations and groups as well as those 
suggested in various articles and publications. This 
search resulted in a total of 45 benchmarks developed 
by these other organizations. Second, institutions that 
have substantial experience in distance education and 
are providing leadership in Internet-based distance 
education were identified. Third, these institutions were 
visited by Institute staff to assess the degree to which 
the campuses incorporated the benchmarks in their 
Internet-based distance learning courses and programs. 
Each site visit included interviews with faculty, 
administrators, and students. These individuals were 
surveyed on both the presence and importance of the 
original benchmarks to determine to what extent they 
are being followed and if they make a difference in terms 
of academic quality. 

The six institutions participating in the study were: 
Brevard Community College, Regents College, the 
University of Illinois at Urbana-Champaign, the 
University of Maryland University College, Utah State 
University, and Weber State University. To qualify for 
selection the institutions (1) must have substantial 
experience in distance education; (2) are recognized 
as among the leaders in distance education; (3) are 
regionally accredited; and (4) offer more than one 
degree program via online distance learning. To ensure 
that a broad spearum of higher education institutions 
were represented, the study included a community 
college, a comprehensive institution, a research 
institution, and a virtual institution. 

The results of the study revealed that, for the most part, 
the benchmarks for quality Internet-based distance 
education were considered important and, in general, 
the institutions strove to incorporate them into their 
policies, practices, and procedures. At the same time, 
several benchmarks did not enjoy consensus among 
administrators, faculty, and students at the institutions 
and, in some instances, were not considered 
mandatory to ensure quality in distance education. 

The following list represents the final benchmarks 
resulting from this study. The Institute's analysis of the 



data and information from the interviews resulted in 
the elimination of 13 benchmarks and the addition 
of three benchmarks. Several benchmarks were 
combined because they addressed the same issue(s) 
and were related to each other. The final outcome is a 
list of 24 benchmarks that are essential to ensure 
quality in Internet-based distance education. These 
benchmarks may assist policymakers — such as college 
and university presidents and chief academic officers, 
state coordinating boards, accrediting bodies, state 
legislatures, and governors' offices — as well as faculty 
and students, in making reasonable and informed 
judgments with regard to the quality of Internet-based 
distance education. 

Institutional Support Benchmarks 

►M A documented technology plan that includes 
electronic security measures (i.e., password 
proteaion, encryption, back-up systems) is in place 
and operational to ensure both quality standards 
and the integrity and validity of information. 

►M The reliability of the technology delivery system is 
as failsafe as possible. 

►M A centralized system provides support for building 
and maintaining the distance education 
infrastructure. 

Course Development Benchmarks 

►M Guidelines regarding minimum standards are used 
for course development, design, and delivery, while 
learning outcomes— not the availability of existing 
technology — determine the technology being used 
to deliver course content. 

►M Instructional materials are reviewed periodically 
to ensure they meet program standards. 

►M Courses are designed to require students to engage 
themselves in analysis, synthesis, and evaluation 
as part of their course and program requirements. 

Teaching/Learning Benchmarks 

►M Student interaction with faculty and other students 
is an essential charaaeristic and is facilitated through 
a variety of ways, including voice-mail and/ore-mail. 

►M Feedback to student assignments and questions is 
constructive and provided in a timely manner. 
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►W Students are instructed in the proper methods of 
effective research, including assessment of the 
validity of resources. 

Course Structure Benchmarks 

►M Before starting an online program, students are 
advised about the program to determine (1) if 
they possess the self-motivation and 
commitment to learn at a distance and (2) if they 
have access to the minimal technology required 
by the course design. 

►M Students are provided with supplemental course 
information that outlines course objectives, 
concepts, and ideas, and learning outcomes for 
each course are summarized in a clearly written, 
straightforward statement. 

►M Students have access to sufficient library resources 
that may include a "virtual library" accessible 
through the World Wide Web. 

►H Faculty and students agree upon expectations 
regarding times for student assignment completion 
and faculty response. 

Student Support Benchmarks 

►M Students receive information about programs, 
including admission requirements, tuition and 
fees, books and supplies, technical and proctoring 
requirements, and student support services. 

►W Students are provided with hands-on training 
and information to aid them in securing material 
through electronic databases, interlibrary 
loans, government archives, news services, and 
other sources. 



►H Throughout the duration of the course/program, 
students have access to technical assistance, 
including detailed instructions regarding the 
electronic media used, practice sessions prior to 
the beginning of the course, and convenient access 
to technical support staff. 

►W Questions directed to student service personnel are 
answered accurately and quickly, with a structured 
system in place to address student complaints. 

Faculty Support Benchmarks 

►W Technical assistance in course development is 
available to faculty, who are encouraged to use it. 

►W Faculty members are assisted in the transition from 
classroom teaching to online instruction and are 
assessed during the process. 

►W Instructor training and assistance, including peer 
mentoring, continues through the progression of 
the online course. 

►W Faculty members are provided with written 
resources to deal with issues arising from student 
use of electronically-accessed data. 

Evaluation and Assessment Benchmarks 

►H The program's educational effectiveness and 
teaching/learning process is assessed through an 
evaluation process that uses several methods and 
applies specific standards. 

►H Data on enrollment, costs, and successful/ 
innovative uses of technology are used to evaluate 
program effectiveness. 

►W Intended learning outcomes are reviewed regularly 
to ensure clarity, utility, and appropriateness. 
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Introduction 



I t has been only 10 years since the coding language 
for the World Wide Web (WWW) was developed 
by Tim Bemers-Lee in Switzerland, and Wide Area 
Information Servers and Gopher protocols became the 
first tools for "surfing the net." Since that time, 
educational institutions, research centers, libraries, 
government agencies, commercial enterprises, 
advocacy groups, and a multitude of individuals have 
rushed to connect to the Internet (Johnson, 1999). It 
seems clear to most observers that the Internet and 
WWW profoundly influence society in general and 
colleges and universities in particular. Not since the 
printing press was invented by Johann Gutenberg in 
the 15th century has an "invention" generated such 
potential to change dramatically how people 
communicate and interact with one another. 

One of the consequences of this tremendous surge in 
online communication has been the rapid growth of 
technology-mediated distance learning at the higher 
education level. A recent survey by the U.S. Department 
of Education's National Center for Education Statistics 
(NCES) found that from 1994-95 to 1997-98 the number 
of distance education degree programs increased by 72 
percent. Moreover, an additional 20 percent of the 
institutions surveyed plan to establish distance education 
programs within the next three years. The survey 
estimated that more than 1.6 million students^ were 
enrolled in distance education courses in 1997-98, a 
number that is surely even higher today (Lewis, 1999). 

This extraordinary growth of technology-mediated 
distance learning in higher education has prompted 
several different organizations to develop principles, 
guidelines, or benchmarks to ensure quality distance 



education.^ These organizations include the American 
Council on Education, the National Education 
Association, the Global Alliance for Transnational 
Education (GATE), the Southern Regional Electronic 
Campus, the Commission on Higher Education of the 
Middle States Association of Colleges and Schools, and 
the Western Cooperative for Educational 
Telecommunications, among others. The quality 
assurance benchmarks promoted by these 
organizations are designed to apply to a wide variety 
of institutional contexts and consist of fairly broad 
statements. Virtually all of the strategies include such 
topics as course development, faculty training, student 
services, learning resources, infrastructure, and 
outcomes assessment. 

These benchmarks, which were developed initially for 
all types of distance learning, have been in existence 
in various forms for a number of years. The question 
that arises is whether they are applicable to Internet- 
based distance education. In short, are the current 
benchmarks appropriate and necessary to ensure 
quality Internet-based distance education? Two 
organizations — the National Education Association 
(NEA), the nation's largest professional association of 
higher education faculty, and Blackboard Inc., a widely 
used platform provider for online education — are 
interested in exploring these issues and their 
implications. As a result, the two organizations jointly 
commissioned The Institute for Higher Education 
Policy to examine the benchmarks by studying active 
distance learning programs at several institutions. The 
Institute was approached by the two commissioning 
organizations in part because of its previous experience 
in analyzing issues related to quality in distance 



* This count is not unduplicated; the survey tallied students in each distance learning course in which they enrolled, not as individual 
students who may have enrolled in one or more distance learning courses. 

^ For the remainder of the report, the term "benchmark" is used to describe the array of principles, strategies, and guidelines that have been 
recommended by the many organizations concerned with quality distance education. In general, a benchmark is an institutional behavior 
that contributes to ensuring quality in technology-mediated distance education. 
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education. The Institute's widely cited 1999 report, 
Whats the Difference! A Review of Contemporary Research 
on the Effectiveness of Distance Learning in Higher 
Education, has generated considerable dialogue 
throughout academia about what constitutes quality 
in distance learning settings. 

Specifically, NEA and Blackboard Inc. asked The 
Institute to attempt to validate those benchmarks that 
have been published by various entities, with specific 
attention to Internet-based distance education. Given 
the rapid growth of Internet-based distance education, 
how realistic are the benchmarks? Do they make sense 
to practitioners of Internet-based distance education? 
Are they taken seriously by higher education 
institutions? To what extent are the benchmarks being 
incorporated into institutions' distance learning 
operations? Are there additional benchmarks that are 
not found in the literature but are being used by 
practitioners that can contribute to the quality of 
distance education? 

This study is designed to answer these questions by 
ascertaining the degree to which the benchmarks are 
aaually being incorporated in the policies, procedures, 
and practices of colleges and universities that are 
distance education leaders. In addition, this case study 
seeks to determine how important the benchmarks are 
to the institutions' faculty, administrators, and students. 
The results of this study can assist policymakers — such 
as college and university presidents and chief academic 
officers, state coordinating boards, accrediting bodies, 
state legislatures, and governors' offices — as well as 
faculty and students, in making reasonable and 
informed judgments with regard to the quality of 
Internet-based distance education. 

A case study addressing quality benchmarks for Internet- 
based distance education inherently includes some of 
the very limitations noted in the What's the Difference! 
report. The major conclusion of the report was that the 
research to date addressing the quality of distance 
education was inconclusive and, thus, much is still 
unknown regarding how, and in what ways, technology 



can enhance the teaching/leaming process (Phipps and 
Merisotis, 1999). This report is not intended to 
overcome many of these limitations. Instead, it uses case 
studies to help build a foundation for future analyses 
capable of refining or expanding upon the lessons 
learned from the institutions studied here. 

Internet-based distance education was a specific focus of 
the case study for at least three reasons. First, Internet- 
based distance education is quickly becoming the 
predominant technology in distance education, which 
is not surprising given the accelerating power of personal 
computers, increasing telecommunications bandwidth 
capabilities, and state-of-the-art software development 
and delivery. The recent NCES report shows that not only 
is Internet-based distance education the most prevalent 
technology, it is also the fastest growing. At least 58 
percent of the reporting institutions who offered distance 
education used Internet-based courses, compared to 54 
percent who used two-way interactive video and 47 
percent who used one-way pre-recorded video. Even as 
the percentage of institutions offering Internet-based 
courses^ jumped from approximately 22 percent to 58 
percent in three years, use of all of the other technologies 
aaually had declined since 1994-95. The report notes 
that Internet-based distance education will be a growing 
avenue for technology-mediated learning in the coming 
years (Lewis, 1999). 

In addition to the Internet's profound influence on 
distance education, it is also important to point out 
that a growing number of faculty are using the Internet 
to complement traditional classroom-based courses. 
For instance, it is not uncommon for course syllabi to 
be placed on the Web. Faculty also are using cyberspace 
to provide access to threaded discussions, group 
activities, and quizzes for their on-campus students. 

Second, Internet-based distance education allows the 
teaching/learning process to occur "at any time and 
any place." The ability to provide asynchronous 
interaaive learning aaivities has become the signature 
charaaeristic of this technology, setting it apart from 
most of the other technologies. Not only does Intemet- 



^ In the 1994-95 survey, this category was entitled computer-based technologies. This narne was changed in 1997-98 to Internet-based courses 
to reflect more accurately the way in which the technologies are used. 
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based instruction allow the teacher and learner to 
communicate over any distance to any place, it alters 
the concept of time. Generally students can participate 
in a course at any time of day or night they choose. 
Indeed, because of this characteristic, the term "24-7" 
has become a part of the lexicon of distance education. 

Third, because of these features, Internet-based 
distance education is, in many ways, fundamentally 
different than traditional classroom-based education. 
Among other things, it is this distinctly different 
concept of time that engenders concern and skepticism 
from many in the higher education community. For 
some, though, it offers exciting new opportunities to 
teach students, as evidenced by the following 
observation from the League for Innovation in the 
Community College: 



The beauty of the Web is that it provides an 
entirely new context for teaching and learning. It 
removes the physical and time constraints for 
instructors as well as learners. Moving a course to 
the Web presents the perfect opportunity to return 
to the core principles of teaching and learning to 
CTeate a new pedagogical model for our practices 
(Boettcher, 1999). 

How can a teaching/learning process that deviates so 
markedly from what has been practiced for hundreds 
of years embody quality education? This question has 
been addressed in many ways by the range of 
benchmarks developed by a variety of organizations 
concerned about quality distance education. These 
benchmarks are examined in more detail in the 
following pages. 
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The Case Study 



T he case study process consisted of three 
sequential phases. First, a comprehensive 
literature search was conducted to compile 
those benchmarks recommended by several 
organizations, as well as those suggested in various 
articles and publications. Second, institutions were 
identified that have substantial experience and are 
providing leadership in distance education. Third, 
these institutions were visited by Institute staff to assess 
the degree to which the campuses incorporated the 
benchmarks in their Internet-based distance learning 
courses and programs. A more detailed explanation 
of each phase of the case study follows. 

Phase I: Review of the Literature 

In addition to reviewing benchmarks that have been 
published by policy and educational organizations, 
several articles by prominent authors in distance 
education were examined. (The selected references 
found at the end of this report provide a better 
understanding of the array of resources reviewed.) 
Considerable overlap existed among the benchmarks 
developed by the various sources and, consequently, 45 
specific benchmarks were identified. These benchmarks 
were grouped into the following seven categories: 

►W Institutional Support. The benchmarks in this category 
include those activities by the institution that help 
to ensure an environment conducive to maintaining 
quality distance education, as well as policies that 
encourage the development of Internet-based 
teaching and learning. These benchmarks address 
technological infrastructure issues, a technology 
plan, and professional incentives for faculty. 

►N Course Development. This category includes 
benchmarks for the development of courseware, 
which is produced largely either by individual 
faculty (or groups of faculty members) on campus. 



subject experts in organizations, and/or 
commercial enterprises. 

►W Teachins/Learning Process. This category addresses 
the array of activities related to pedagogy, the art 
of teaching. Included in this category are process 
benchmarks involving interaaivity, collaboration, 
and modular learning. 

►W Course Structure. The benchmarks in this category 
address those policies and procedures that support 
and relate to the teaching/leaming process. They 
include course objectives, availability of library 
resources, types of materials provided to students, 
response time to students, and student expectations. 

►W Student Support. This category includes the array 
of student services normally found on a college 
campus including admissions, financial aid, etc. — 
as well as student training and assistance while 
using the Internet. 

►M Faculty Support. Since not every faculty member 
possesses the skills and temperament for Internet- 
based distance learning, these benchmarks 
address activities that assist faculty in teaching 
online, including policies for faculty transition 
help as well as continuing assistance throughout 
the teaching period. 

►W Evaluation and Assessment.'^ The benchmarks in this 
category relate to policies and procedures that 
address how, or if, the institution evaluates 
Internet-based distance learning. They include 
outcomes assessment and data collection. 

Phase II: Identification of Institutions 

Sbc institutions were identified for site visits. To qualify 

for selection the institutions (1) must have substantial 

experience in distance education; (2) are recognized as 

among the leaders in distance education; (3) are 



* Placing the benchmarks in these seven categories is not the only way to dassify them. Others may use, and have indeed used, other categories. 
The categories presented here reflect a simple logic that addresses major functions of any institution. 
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regionally accredited; and (4) offer more than one 
degree program via online distance learning. To ensure 
that a broad spectrum of higher education institutions 
were represented, the case study included a community 
college, a comprehensive institution, a research 
institution, and a virtual institution. The following 
institutions participated in the case study (a detailed 
profile of each institution is provided in Appendix One): 

►H Brevard Community College. This Florida college began 
offering distance education courses in 1974 and is 
also a participant in the PBS Adult Learning Service 
"Going the Distance Program." Brevard received a 
grant from the U.S. Department of Education s Fund 
for the Improvement of Postsecondary Education for 
faculty training and development in order to 
positively impact student learning. 

►W Regents College. This unique institution began in 
1971 as the External Degree Program of the 
University of the State of New York. Focusing on 
working adults, the college offers online graduate 
programs in Nursing and Liberal Studies. Over 
83,000 people worldwide have earned a college 
degree through Regents' programs. 

►W University of Illinois at Urbana-Champaign.^ As one 
of three participants in the University of Illinois 
Online, the institution offers 20 degree- or 
certificate-programs over the Internet, with more 
in development. The university received a grant 
from the Pew Grant Program of Course Redesign 
in 1999 to redesign a two-course sequence in 
Economic Statistics. 

►M University of Maryland University College. This virtual 
institution was founded with the mission of 
providing continuing education to Maryland's 
professional workforce. With over 25 years 
experience in distance education, the college 
offers 14 bachelor's and 10 master's degree programs 
online. The college has twice been awarded the 



University Continuing Education Association's 
Award for Innovative Distance Education. 

►M Utah State University. This institution has been 
involved in various forms of distance education 
since 1 911 . The university offers two types of online 
courses, semester-based and independent study, 
and has been selected as a participant in the federal 
Learning Anytime Anywhere Partnerships program 
created in the 1998 reauthorization of the Higher 
Education Act. 

►M Weber State University (WSU). This comprehensive 
institution launched WSU Online in 1997 and 
currently offers two-thirds of the online learning 
courses in Utah. WSU received an Innovation in 
Distance Education Award from the University 
Continuing Education Association in 1998. 

Phase III: The Institutional Visit 

Between September 1999 and January 2000, Institute 
staff visited each institution. In addition to 
conducting indepth interviews with faculty, 
administrators, and students, a survey using a Likert 
Scale^ was administered to each interviewee.^ The 
survey also was administered to distance education 
students who were not able to participate in the 
interview process primarily because they did not 
reside near the institution. In all, 27 faculty, 62 
administrators, 16 individuals who were both a 
faculty member and an administrator, and 42 
students were interviewed and/or completed a survey, 
for a total of 147 respondents. 

The Likert Scale listed the 45 benchmarks and 
requested each person to rank each benchmark on two 
criteria. First, to what extent is the benchmark true for 
the distance learning program (ranked from 
l=completely absent to 7=completely present)? 
Second, how important is each benchmark to ensure 



® Unlike the other colleges and universities in the case study where the entire institution participated in the case study, at the University of 
Illinois at Urbana-Champaign, only two Internet-based programs were reviewed: Master's programs in Human Resource Development and 
Curriculum, Technology, and Education Reform, 

^ A Likert scale is an instrument that asks an individual to respond to a series of statements by indicating whether he or she agrees or 
disagrees with each statement. 

^ The authors want to express their appreciation to the faculty, administrators, and students who were interviewed. We were struck by the 
knowledge and impressive professionalism of the faculty and administrators involved in distance learning, and the motivation and excitement 
of the students. The level of enthusiasm for teaching was extraordinary and the willingness of the faculty and administrators to share both 
their successes and challenges revealed a candor that enhanced the credibility of this study. 
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quality (ranked from l=not important to 5=very 
important)? Those persons who did not have sufficient 
knowledge or experience relating to the benchmark 
could check a "Not Applicable" box. The mean score, 
mode, and standard deviation were computed for each 
benchmark (see Appendix Two). 

At the extremes, this process could result in four 
hypothetical scenarios: a benchmark could be present 
completely and very important; a benchmark could 
be present completely and not important; a 
benchmark could be absent completely and very 
important; and a benchmark could be absent 
completely and not important. The actual results are 
provided in the next section. 

Benchmarks Identified from the Literature 
Institutional Support 

1. Faculty are provided professional incentives for 
innovative practices to encourage development of 
distance learning courses. 

2. There are institutional rewards for the effective 
teaching of distance learning courses. 

3. A documented technology plan is in place to 
ensure quality standards. 

4. Electronic security measures are in place to ensure 
the integrity and validity of information. 

5 . Support for building and maintaining the distance 
education infrastructure is addressed by a 
centralized system. 

Course Development 

6. Distance learning course development must be 
approved through a broad peer review process. 

7. Guidelines exist regarding minimum standards for 
course development, design, and delivery. 

8. Course design is managed by teams comprised of 
faculty, content experts, instructional designers, 
technical experts, and evaluation personnel. 

9. During course development, the various learning 
styles of students are considered. 

1 0. Assessment instruments are used to ascertain the 
specific learning styles of students, which then 
determine the type of course delivery. 



1 1. Courses are designed with a consistent structure, 
easily discemable to students of varying learning 
styles. 

12. The technology being used to deliver course 
content is based on learning outcomes. 

13. Instructional materials are reviewed periodically 
to ensure they meet program standards. 

Teaching/Leaming Process 

14. Student interaction with faculty is facilitated 
through a variety of ways. 

15. Student interaction with other students is 
facilitated through a variety of ways. 

1 6. Feedback to student assignments and questions is 
provided in a timely manner. 

17. Feedback to students is provided in a manner that 
is constructive and non-threatening. 

1 8. Courses are separated into self-contained segments 
(modules) that can be used to assess student 
mastery before moving forward in the course or 
program. 

19. The modules/segments are of varying lengths 
determined by the complexity of learning 
outcomes. 

20. Each module/segment requires students to engage 
themselves in analysis, synthesis, and evaluation 
as part of their course assignments. 

21. Class voice-mail and/or e-mail systems are 
provided to encourage students to work with each 
other and their instruaor(s). 

22. Courses are designed to require students to work 
in groups utilizing problem-solving activities in 
order to develop topic understanding. 

23. Course materials promote collaboration among 
students. 

Course Structure 

24. Students are provided with supplemental course 
information that outlines course objectives, 
concepts, and ideas. 

25. Specific expectations are set for students with 
respect to a minimum amount of time per week 
for study and homework assignments. 
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26. Faculty are required to grade and return all 
assignments within a certain time period. 

27. Sufficient library resources are made available to 
the students. 

28. Students are instructed in the proper methods of 
effective research, including assessment of resource 
validity. 

29. Before starting the program, students are advised 
about the program to determine if they have the self- 
motivation and commitment to learn at a distance. 

30. Learning outcomes for each course are summarized 
in a clearly written, straightforward statement. 

Student Support 

31. Students can obtain assistance to help them use 
electronically accessed data successfully. 

32. Students are provided with hands-on training and 
information to aid them in securing material 
through electronic databases, interlibrary loans, 
government archives, news services, etc. 

33. Written information is supplied to the student 
about the program. 

34. Easily accessible technical assistance is available 
to all students throughout the duration of the 
course/program. 

35. A structured system is in place to address student 
complaints. 



Faculty Support 

36. Technical assistance in course development is 
available to faculty and they are encouraged to use 
it. 

3 7 . Faculty members are assisted in the transition from 
classroom teaching to distance instruction and are 
assessed in the process. 

38. There are peer mentoring resources available to 
faculty members teaching distance courses. 

39. Distance instructor training continues throughout 
the progression of the online class. 

40. Faculty members are provided with written 
resources to deal with issues arising from student 
use of electronically-accessed data. 

Evaluation and Assessment 

41. The program's educational effectiveness is 
measured using several methods. 

42. An evaluation process is used to improve the 
teaching/learning process. 

43. Specific standards are in place to compare and 
improve learning outcomes. 

44. Data on enrollment, costs, and successful/ 
innovative uses of technology are used to evaluate 
program effectiveness. 

45. Intended learning outcomes are regularly reviewed 
to ensure clarity, utility, and appropriateness. 
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How Did the Benchmarks Rate? 



I t was clear from the site visits that quality 
benchmarks were considered with great care and 
embraced by every institution that participated in 
the case study. At every campus, at least one faculty 
member, administrator, or often several students stated 
that the benchmarks were on target and (with some 
exceptions that will be explained later in the report) 
need to be addressed to ensure quality in Internet- 
based distance education courses and programs. As one 
respondent said about the survey, "Good questions. 
While the 'best' possible program would have rated all 
of these items as very important and strongly agree, 
one must consider resources and growing pains so as 
to target the most important areas." 

In addition, several respondents were somewhat 
chagrined that Internet-based distance education was 
under such intense scrutiny while the traditional 
classroom-based teaching was not under the same 
spotlight. One person showed her exasperation by 
writing, "The same standards and benchmarks we use 
in distance education should be applied to on-campus 
courses as well. It doesn't happen that way here or 
across the country." 

It should be noted that a// of the institutions are included 
in the data presented in this report. Because the 
intention of this case study is to validate the benchmarks 
for the higher education community in general, it serves 
no purpose to separate the data for each institution. 
While there are exceptions to the conclusions and 
comments from the interviews, the following discussion 
represents a consensus of a majority of the institutions 
in the case study. It is, therefore, not appropriate to 
assume that the attributes outlined in the discussion 
always represent each and every institution. 

The information for this case study is both quantitative 
(the survey using a Likert Scale) and qualitative (the 
indepth interviews). No effort was made to apply any 

O 

ERIC 



statistical tests to ascertain the degree of importance 
of a benchmark and its presence at the institutions, 
and the difference between the two. Instead, the more 
than 100 interviews (all lasting at least an hour) guided 
the analysis. The respondents were interviewed after 
completing the survey and many of their comments 
were directed to specific items in the survey. The 
interviews informed the quantitative data and served 
to put a "human face" on the stark numbers. The 
standard deviations of each benchmark are presented 
in Appendix Two to demonstrate more fully the range 
of consensus for each item. 



This section is organized around the seven categories 
of benchmarks: Institutional Support, Course 
Development, Teaching/Learning Process, Course 
Structure, Student Support, Faculty Support, and 
Evaluation and Assessment. Each category is 
accompanied by a chart showing the relationship 
between the degree of presence for the benchmark and 
its importance to the institution. The discussion of the 
category includes comments from the interviews. 



Institutional Support 

All of the benchmarks in this category were considered 
important to ensure quality Internet-based distance 
education. The benchmark addressing electronic 
security measures (#4) received exceptionally high 
ratings both with regard 



to importance 
presence at 



and 
the 

institution. There was a 
marked difference 
between the importance 
of the benchmark 
regarding a documented 



When contemplating the 
effect of the Internet on 
higher education, we better 
understand what 7^ graders 
are doing — because they are 
the freshman class of 2005. 
—ADMINISTRATOR 



technology plan (#3) 

and the actual presence of a plan on campus. 
Respondents at some institutions complained that 
because of the extraordinary growth of Internet-based 
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Fisure One: Institutional Support Benchmarks 



Not Important importance Very Important 
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1. Faculty are provided professional 
incentives for innovative practices 
to encourage development of 
distance learning courses. 



2. There are institutional 
rewards for the effective teaching 
of distance learning courses. 



3. A documented technology 
plan is in place to ensure 
quality standards. 



4. Electronic security measures 
are in place to ensure the 
integrity and validity of information. 



5. Support for building and 
maintaining the distance 
education infrastructure is 
addressed by a centralized system. 
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Strongly Disagree presence Strongly Agree 



Importance Presence 



distance education, infrastructure is constantly trying 
to keep pace with demand. 

Compelling student demand has other consequences 
with regard to planning. Because of increasing student 
interest in Internet-based distance education at some 
of the institutions in the case study, administrators 
revealed that policies are being developed to catch up 
with practice. One administrator said simply that the 
institution is moving ahead without all of the answers. 
While some institutions were farther ahead in their 
planning than others, some institutions that are 
struggling to keep up with the demand for Internet- 
based courses have made a conscious decision to serve 
students immediately and plan later. 

Figure One reveals a substantial difference between the 
importance of the benchmarks concerning professional 
incentives and institutional rewards (#1 and 2) and their 
presence on campus. While considered very important, 
rewarding and providing incentives to faculty for good 



online teaching did not receive high 
scores with regard to their presence at the 
institutions. This conclusion was 
reinforced in the interviews, as revealed 
by comments from one faculty member: 
"We operate on a 'shoestring,' and the 
tenure, promotion, and reward system 
does not take into account the extra time 
a faculty member needs to teach well in 
distance education courses." Another 
administrator noted that at his institution, 
distance education "has some of our 
better instructors even though the rewards 
have not been there for them." 

The dedication and knowledge of the 
faculty who teach Internet-based distance 
education was apparent in the structured 
interviews. The faculty who were 
interviewed showed keen insights into the 
teaching/leaming process, and many were 
considered by their peers as some of the 
best teachers at the institution. 



The high number of excellent faculty 
teaching Internet-based distance 
education courses can be explained partially by the 
faculty selection practice of many of the institutions. 
With few exceptions, faculty volunteered to teach 
distance education courses. One institution used the 
phrase "work with the willing" to explain this faculty 
selection process. Faculty who came forward, 
irrespective of the course they wished to teach, were 
often allowed to develop a course. Frequently these 
faculty were veteran teachers who were eager to teach 
using the Internet. This selection process, of course, 
produces an array of courses that may not necessarily 
represent a strategic plan of course and programmatic 
offerings. This is not to suggest, however, that Internet 
courses are generated without planning or strategy. 
As one administrator put it, "...although this 
institution did not have a strategic plan, 'strategic 
thinking' was used in the development of Internet 
courses and programs." For instance, several 
institutions attempted to develop Internet offerings 
for courses that had high enrollments such as lower- 
division, general education courses. 
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Many respondents suggested that a new benchmark 
should be added to the list: "The reliability of the 
technology delivery system should be as failsafe as 
possible." They explained its importance this way: "If 
the lights go out in a traditional classroom, it may be 
an inconvenience. If the system crashes, if s a calamity! " 

Course Development 

The three benchmarks relating to learning styles received 
a mixed reaction from the respondents. As shown in 
Figure Two, benchmarks 9 and 11 were not considered 
very important and did not have a high degree of 
presence at the institutions. Benchmark 10 received low 
scores with respea to both importance and presence. 
The interviews helped to explain these sentiments. Many 
faculty and administrators stated that while these 
learning style benchmarks sound good on paper, 
actually meeting the standard is very difficult. One 
faculty member suggested that the research on learning 
styles is very poor and simply may not be an appropriate 
criterion for benchmarks. 



degree of presence. There was considerable concern 
regarding the meaning of "broad." Many respondents 
made itveryclearthat peer review is, and should be, similar 
to the traditional model primarily at the departmental 
level. As one respondent explained, "Quality control is a 
department responsibility; additional requirements may 
endanger academic freedom 
and shared governance." 



The lines between 
traditional classroom 
education and Internet- 
based education are 
becoming blurred. 
—ADMINISTRATOR 



It is important to note that a 
possible disparity exists 
between the technology that an 
institution might possess and 
the technology available to the typical student. Although 
institutions may have enhanced, or are enhancing, their 
capability with high speed networks with additional 
bandwidth, and improved video quality, course 
development must take into consideration the technology 
that the students possess. Many students have older 
personal computers, equipped with slow modems, 
insufficient memory, and small hard drives. Put more 



The benchmark addressing the 
management of course design (#8), 
also received a mixed reaaion. By 
and large, the development of a 
course was the responsibility of 
individual academic departments 
and the extent to which the 
procedure met the criteria in the 
benchmark was a departmental 
decision. One faculty member said, 
"Some of the items suggest that 
greater evaluation, oversight, and 
course development ought to take 
place in distance education than for 
traditionally delivered courses. I 
don't buy this assumption. 
Traditional courses rarely come 
under that kind of scrutiny on this 
campus or others." 

The notion of broad peer review 
generated many comments. The 
benchmark on the approval of course 
development (#6) was not considered 
very important and did not show a high 



Figure Two: Course Develoment Benchmarks 



Not Important importance Very Important 



6. Distance learning course 
development must be approved 
through a broad peer review process. 

7. Guidelines exist regarding 
minimum standards for course 
development, design, and delivery. 

8. Course design is managed by teams comprised of 
faculty, content experts, instructional designers, 
technical experts, and evaluation personnel. 

9. During course development 
the various learning styles 
of students are considered. 

10. Assessment instruments are used to ascertain 
the specific learning styles of students, which 
then determine the type of course delivery. 

11. Courses are designed with a 
consistent structure, easily discernable to 
students of varying learning styles. 

12. The technology being used to 
deliver course content is based 
on learning outcomes. 

13. Instructional materials are 
reviewed periodically to ensure 
they meet program standards. 
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bluntly by one administrator, "Bells and whistles have to 
be available and understood by both the teacher and the 
student." The disparity between the technological 
capabilities of the institution and its students was referred 
to as "cutting edge, not bleeding edge" by one interviewee, 
while another referred to the "trailing edge" of technology. 

Teachins/Learning Process 

The majority of the benchmarks regarding the teaching/ 
learning process were considered both important and 
present on the campus. However, 
as shown in Figure Three, the 
benchmarks related to 
collaboration and modular 
learning were not endorsed 
widely. Benchmarks 22 and 23 
were not seen to be very important 
and, likewise, did not display as high of a degree of 
presence at the institutions as 
other benchmarks in this 
category. Several comments 
may help to explain this. In 
general, faculty and 
administrators suggested that 
collaboration as an end in 
itself was not necessary. 

Collaboration depends upon 
factors such as the content of 
the course, the level of 
instruction (undergraduate 
versus graduate), and the 
values of the instructor. 

Regardless of its importance, 
however, one faculty member 
noted that online students 
are collaborating much 
more than students in 
traditional classes. 

With respect to modular 
learning, several respondents 
did not consider these 
benchmarks very relevant. 

Benchmarks 18 and 19 are, as 
one faculty member wrote, 

"...naive and passe. In several 
respects, we have gone 



beyond that." Another stated that modular learning is 
not supported by the research and is not appropriate as 
a stand-alone benchmark. 

A Word About Interactivity 

Figure Three shows three benchmarks that address the 
process of interactivity (#14, #15, and #16), all of 
which received high scores for importance and 
presence. It has become increasingly evident that 
interactivity is the sine cjua non for quality in distance 
learning. Indeed, many would say that it is crucial for 
any type of learning. As Otto Peters, author of L^amin^ 
and Teaching in Distance Education, writes: 

If we take distance education seriously and 
understand it to be something more than the 
mere distribution and reading of study materials, 
we must provide sufficient opportunities for 



Figure Three: Teaching/Learning Process Benchmarks 
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Very Important 



14. Student interaction with 
faculty is facilitated through 

a variety of ways. 

15. Student interaction with 
other students is facilitated 

through a variety of ways. 

1 6. Feedback to student 
assignments and questions 
is provided in a timely manner. 

' 1 7. Feedback to students 

is provided in a manner that 
is constructive and non-threatening. 

18. Courses are separated into self-contained 
segments (modules) that can be used to assess student 
mastery before moving forward in the course or program. 

19. The modules/segments are of 
varying lengths determined by 
the complexity of learning outcomes. 

20. Each module/segment requires students to 
engage themselves in analysis, synthesis, and 
evaluation as part of their course assignments. 

21. Class voice-mail and/or e-mail systems 
are provided to encourage students to 
work with each other and their instructor(s). 

22. Course are designed to require students 
to work in groups utilizing problem solving 
aaivities in order to develop topic understanding. 

23. Course materials promote 
collaboration among students. 
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Online courses require 
more discipline, but 
you can work at your 
own pace. 
—STUDENT 
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dialogues. If, in addition, we understand 
academic studies as a process in which the aim is 
education through knowledge, we cannot do 
without a considerable proportion of dialogical 
learning and teaching in distance education. 
(Peters, 1999, pg. 39) 

Many respondents agreed. One faculty member wrote 
that "Students need to communicate almost on a daily 
basis, not just once a week. " Another noted that "in order 
to facilitate interactivity, in addition to e-mail and voice- 
mail, a conference system is very important." One student 
refleaed the observations of several others in stating, "The 
majority of online classes I have taken have been set up 
very well for a successful educational experience. The 
instruaors for these classes encouraged communication 
between their students and themselves and were available 
to answer questions at any time through e-mail. They 

Figure Four: Course Structure Benchmarks 

Not Important 



also provided timely and valuable feedback on 
assignments, which is very important for a students 
academic success." 



of 



Self-directed study, which is 
prevalent at the graduate level, 
is being pushed down to the 
undergraduate level because of 
online learning. 
—ADMINISTRATOR 



IMPORTANCE 



24. Students are provided with supplemental 
course information that outlines course 
objectives, concepts, and ideas. 

25. Specific expectations are set for students 
with respect to a minimum amount of time 
per week for study and homework assignments. 

26. Faculty are required to grade 
and return all assignments 
within a certain time period. 



27. Sufficient library resources are 
made available to the students. 



28. Students are instructed in the 
proper methods of effective research, 
including assessment of resource validity. 



29. Before starting the program, students are advised 
about the program to determine if they have the self- 
motivation and commitment to learn at a distance. 



30. Learning outcomes for each 
course are summarized in a clearly 
written, straightforward statement. 
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The notion 
interactivity is 
highlighted here, not 
only because it is 
central to the quality 
of distance learning, 

but also because it leads to the realization that Internet- 
based distance education is evolving its own pedagogy. 
As noted by the League for Innovation in the 
Community College and others, online courses have 
certain characteristics that are unique to the 
technology — which allows the exploration of new 
pedagogical models. Faculty are still learning about, 
and experimenting with, different ways that students 

can interact with faculty, 
other students, and a wide 
variety of instructional 
resources. One institution 
combines the use of 
asychronous technologies 
to facilitate much of the 
classroom interactions 
and synchronous 

communication to 
facilitate small group 
interactions and course 
office hours. With respect 
to interactivity, one faculty 
member lamented that, 
"Too often we try to 
emulate the classroom. 
Even software vendors try 
to emulate the classroom. 
The fact is that online 
learning is affecting how 
we teach in traditional 
classrooms." 



Very Important 
5 



Stronsly Disasree 



presence 
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Course Structure 

In general, the course 
structure benchmarks were 
seen as both important and 
present at the institutions. 
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However, as illustrated in Figure Four, benchmarks 25 
and 26 addressing specific time requirements were the 
exception, scoring lower in terms of presence. Many 
respondents balked at the prescriptive nature of these 

benchmarks. Given the 



Students need to understand 
more fully the role and 
responsibility they need to 
participate in online courses. 
—ADMINISTRATOR 



dynamic and innovative 
characteristics of Internet- 
based distance education, 
particularly the capacity 
for students to pace 
themselves in a variety of 
ways, hard and fast rules on how much work should 
be accomplished in a specific time period or the 
precise response time for a faculty member 
is inappropriate. One student responded to a minimum 
amount of time per week for study: "Yeah, they (faculty) 
give us recommendations, but they don't 'push' you. 
That's good! I enrolled in a [online] course so I would 
have the freedom to study at my pace and when I wanted 
to study. I did everything at my own pace for the first 
course and I got an 'A.' Therefore, stressing a strict pace is 
'not important.'" However, there was strong consensus 
that faculty and students must 
agree when assignments need to 
be completed and returned. 



The very high ratings for the 
benchmark regarding library 
resources (#27) are worth 
noting. In many respects, the 
library has come to the 
student's home or workplace. 
At several institutions, the 
librarian scans any article the 
faculty member requests, 
providing the full text online. 
In addition, the library 
resources include many 
online full-text books and, if 
needed, interlibrary loans. 
Reference assistance to 
students is not neglected and 
some respondents suggested 
that a reference person 
should be available 24 hours 
a day, 7 days a week. At least 
one institution required 



students to take a four-hour, non-credit course about 
library resources. 

It may be helpful to examine briefly the issue of class 
size, although there are no benchmarks directly 
related to maximum class size. Because the case study 
revealed a wide variety of opinions, it is inappropriate 
that a specific benchmark on maximum class size be 
introduced. More than one respondent suggested that 
there should be a maximum size, say 20 or 25 
students. Another recommended that the /irst online 
course a faculty member teaches should enroll no 
more than 15 students. Yet another institution is 
experimenting with class size and is incrementally 
increasing student enrollment in Internet courses. At 
the other extreme, one institution offers a two-credit 
orientation course with over 300 students enrolled. 
When asked how much of the faculty time per week 
was required for the large enrollment course, the 
answer was about 20 hours. The instructor noted that 
appropriate interaction and successful student 
outcomes can be achieved in courses with large 



Figure Five: Student Support Benchmarks 
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31. Students can obtain assistance 
to help them use electronically 
accessed data successfully. 



32. Students are provided with hands-on training 
and information to aid them in securing 
material through electronic databases, interlibrary 
loans, government archives, news services, etc. 



33. Written information is supplied 
to the student about the program. 



34. Easily accessible technical assistance 
is available to all students throughout 
the duration of the course/program. 



35. A structured system is in place 
to address student complaints. 
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enrollments using a variety of methods, including 
student collaboration. It could be argued that 
maximum class size relates more to faculty course 
workload than student outcomes. It appears, 
therefore, that a specific benchmark for class size is 
ill advised, and much more experimentation needs 
to be conducted. 

Student Support 

Each of the benchmarks in this category were ranked 
as having a high degree of importance. However, it 
appears that, for at least three benchmarks concerning 
technical assistance and training for students (#31, 
32, and 34), the institutions feel they have some way 
to go. 

It is important to understand that traditional on- 
campus students are among those participating in 
Internet-based distance education, taking one or two 
online courses because it is convenient or to avoid 
conflicts with another course. Therefore, student 
support services provide for a broader audience than 



would otherwise be 
expected. Furthermore, it 
was noted that those 
students who are at a 
distance and do not 
come to the campus may 
have greater need for 
feedback than the 
traditional student. As 
one student noted, 'T 
find that online advisement 
than course work is the real 



/ have only had two bad 
experiences tuith online classes, 
mainly because the course was 
not set up as completely or as 
well as other courses. They were 
more like correspondence courses 
and lacked feedback from the 
professors on assignments. 
—STUDENT 

and school services other 

problem." 



To address this problem, one institution has 
established an electronic peer network in a web-based 
environment that enables students to interact 
academically and socially online. Students are able 
to identify other students with common interests, 
participate in live chats and threaded discussion 
groups, exchange books and study materials, locate 
study partners, access career resources, and/or join 
an online study group. 



Figure Six: Faculty Support Benchmarks 
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36. Technical assistance in course 
development Is available to faculty 
and they are encouraged to use it. 



37. Faculty members are assisted In the 
transition from classroom teaching to distance 
instruction and are assessed in the process. 



38. There are peer mentoring 
resources available to faculty 
members teaching distance courses. 



39. Distance instructor training 
continues throughout the 
progression of the online class. 



40. Faculty members are provided with 
written resources to deal with issues arising 
from student use of electronically-accessed data. 
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Many respondents recognized 
that a number of students need 
more preparation for working 
on the Internet. As a 
consequence, institutions 
provide technical help through 
a variety of ways, including an 
800 number, e-mail, a real-time 
chat room, and an online 
tutorial for technical assistance. 
Technical staff meet several 
times a year to improve the 
technical support based on 
student complaints. 

In areas other than technical 
assistance, student support 
personnel are devising several 
ways to help online students. 
One institution has set the goal 
of "one-stop shopping." 
To accomplish this goal, 
advisors have information 
on a computer screen that 
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covers virtually all of a particular student's needs, 
including financial aid, academic program, tuition and 
fees, etc. At another institution, an advisor saw herself 
as a "coach," providing both support and 
encouragement for online students. These issues led 
several respondents to suggest that a new benchmark 
needs to be included under student support: 
"Questions directed to student service personnel must 
be answered accurately and quickly." 



Figuring out what works is 
an exciting challenge in 
education, but magnified 
by Internet-based distance 
education. It's an exciting 
time to be an educator, 
—FACULTY 



Faculty Support 

With the possible exception 
of the benchmark addressing 
written resources for faculty 
on student use of 
electronically-accessed data 
(#40), the rest of the 
benchmarks were considered very important. However, 
the three benchmarks concerning training and 
technical assistance for faculty (#37, 38, and 39) had 
lower scores for presence. 



lamented that the technical aspect of online teaching 
is sometimes overwhelming. Another faculty member 
wrote that "pedagogy of online learning must be part 
of training and the online environment." 

Evaluation and Assessment 

All of the benchmarks in this category are considered 
important by the institutions in the case study. 
Benchmark 43 addressing specific standards, however, 
had a relatively low mark for presence. By and large, all 
of the institutions had systems in place that addressed 
evaluation and assessment. More than one institution 
had elaborate evaluation and assessment procedures. It 
is clear that all of the institutions recognize the 
importance of evaluation and assessment. 

One interesting evaluation system for online 
instruction that is still evolving invokes a medical 
analogy in which a physician examines a patient's vital 
signs to determine the patient's current state of health. 
Vital signs that are below acceptable standards are 



Every institution in the case study 
has systematic processes for 
transitioning faculty to online 
instruction from the traditional 
classroom and for training and 
assisting faculty teaching online 
courses. Given that a substantial 
number of faculty teaching online 
courses volunteered for the 
assignment, and many are veteran 
teachers respected by their peers, 
their standards are quite high and 
their dedication is keen. 
Information from the interviews 
strongly suggested that a major 
reason for the less than optimum 
presence of assistance was lack of 
resources, not lack of will. As noted 
earlier, several of the campuses are 
trying to keep up with student 
demand, while sufficient personnel 
and financial resources may be 
lagging behind. This is particularly 
true in the area of technical 
assistance. One faculty member 



Figure Seven: Evaluation and Assessment Benchmarks 
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41. The program’s educational 
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using several methods. 



42. An evaluation process is 
used to improve the 
teaching/leaming process. 



43. Specific standards are in 
place to compare and 
improve learning outcomes. 



44. Data on enrollment^ costs, and 
successful/innovative uses of technology 
are used to evaluate program effectiveness. 



45. Intended learning outcomes are 
regularly reviewed to ensure 
clarity, utility, and appropriateness. 
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examined in more detail; utilizing more precise 
information and investigative techniques. In distance 
education; data are used to calculate a program's 
"health" rating in six areas: student demand; student 
retention; student satisfaction; faculty satisfaction; 
student achievement; and financial efficiency. 

One issue that was mentioned by more than one 
respondent had to do more with outcomes; 
specifically student attrition. In The Institute for 
Higher Education Policy's 1999 report; What's the 
Difference? A Review of Contemporary Research on the 
Effectiveness of Distance Learning in Higher Education, 
concern regarding the apparent high attrition of 



online students as compared to traditional students 
was highlighted (Phipps and Merisotis, 1999). One 
administrator mentioned that there seems to be an 
inordinately high number of the following grades: 
"A;" "F;" and/or "W" (withdraw) for online students. 
Respondents from other institutions in the case 
study also referred to this problem. This anecdotal 
evidence suggests that there may be a bipolar 
distribution where students are either quite 
successful or dropping out. This further supports the 
conclusion in What's the Difference? that student 
attrition in Internet-based distance education 
courses is an important research topic in the 
evaluation and assessment programs of institutions. 
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Recommendations 



T his case study of six higher education institutions 
revealed that, for the most part, the benchmarks 
for quality Internet-based distance education 
were considered important and, in general, the 
institutions strove to incorporate them into their 
policies, practices, and procedures. At the same time, 
there were several benchmarks that did not enjoy 
consensus among the administrators, faculty, and 
students at the institutions and, in some instances, 
were not considered mandatory to ensure quality 
distance education. 

The purpose of this case study is to help 
policymakers make judgments with regard to quality 
Internet-based distance education. The challenge, 
then, is to identify those benchmarks that are 
essential for quality distance education, in contrast 
to those benchmarks that contribute to and support 
the teaching/learning process, but are not necessary 
or required to ensure quality. 

Analysis of the quality indicators at these institutions 
suggests that the benchmarks that are mandatory are 
those for which the absence of the benchmark would detract 
from quality. Stated positively, the benchmark must be 
essential or imperative to ensure quality in Internet- 
based distance education. Our recommendations are 
based upon the quantitative data from the surveys but 
are moderated by the hours of interviews with faculty, 
administrators, and students. Even though a particular 
benchmark may have received a high score for either 
presence and importance, the interviews informed and 
tempered our understanding. 

The Institute s analysis of the data and information 
from the interviews resulted in the elimination of 13 
benchmarks and the addition of 3 benchmarks. 
Moreover, several benchmarks were combined because 
they addressed the same issue(s) and were related to 
each other. The final outcome is a list of 24 benchmarks 



that are essential to ensure quality in Internet-based 
distance education. 

Benchmarks That Are Not Essential 

The following benchmarks are not essential for 
ensuring quality Internet-based distance education. 

Institutional Support Benchmarks 

►W Faculty are provided professional incentives for 
innovative practices to encourage development of 
distance learning courses (^1). 

►W There are institutional rewards for the effective teaching 
of distance learning courses (#2). 

This recommendation may be the most controversial. 
Both of these benchmarks enjoyed consensus with 
regard to their importance to ensuring quality. Yet, 
despite their relatively low presence at the institution, 
quality Internet-based distance education was 
occurring at every one of these institutions. As noted 
earlier, the faculty involved in distance education were 
dedicated, experienced, enthusiastic, and exhibited a 
high degree of professionalism. Although they 
preferred to receive professional incentives and 
institutional rewards, the vast majority of the faculty 
engaged in distance learning because they were excited 
about it, they found it intrinsically rewarding, and they 
were already recognized by their peers as good teachers. 
Innovative distance education is happening every day 
without this benchmark. Moreover, many interviewees 
noted that distance education should be treated no 
differently than traditional classroom-based teaching. 

Course Development Benchmarks 

►W During course development the various learning styles 
of students are considered (^9). 

►W Assessment instruments are used to ascertain the specific 
learning styles of students, which then determine the 
type of course delivery (^10). 
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►W Courses are designed with a consistent structure, easily 
discemable to students of varying learning styles (#11 j. 

The benchmarks addressing student learning styles 
received a cool reception from many faculty and 
administrators. In the view of several respondents, 
benchmarks addressing student learning styles were often 
platitudes with litde basis in research and were very difficult 
to accomplish. While there is an implicit recognition of 
how students learn and an explicit understanding of the 
importance of interaction, constmctive feedback, and other 
characteristics of good pedagogy, benchmarks that required 
these practices are not necessary to ensure quality 

►M Course design is managed by teams comprised of faculty, 
content experts, instructional designers, technical 
experts, and evaluation personnel (#S). 

This benchmark was considered by many to be overkill. 
By and large. Internet-based distance education is the 
responsibility of the instructor and the academic 
department. The course development practices varied 
from department to department and institution to 
institution. The degree to which teams were formed 
for course design depended upon many factors, 
including the experience and knowledge of the faculty 
member, the degree to which the instructor asked for 
assistance, and departmental policies. 

►M Distance learning course development must be approved 
through a broad peer review process (#6), 

The majority of faculty and administrators objected 
strongly to this benchmark, particularly expressing 
concern about the definition of "broad." Peer review 
for distance education courses should be no different 
than that for traditional campus-based courses was a 
common refrain. In addition to concerns about 
academic freedom, subjecting distance learning courses 
to procedures that are more rigorous than the usual 
campus policies suggested a degree of oversight that was 
both unnecessary and detrimental to the ability of 
faculty to be innovative and responsive to student needs. 

Teaching/Leaming Benchmarks 

►H Courses are designed to require students to work in 
groups utilizing problem-solving activities in order to 
develop topic understanding (#22). 



►M Course materials promote collaboration among 
students (#23). 

The benchmark addressing the importance of 
collaboration was not viewed as essential to quality 
Internet-based distance education. It is important to 
note that many courses incorporated collaboration 
among students and some instructors even required 
it. However, including this benchmark as mandatory 
to ensure quality in distance education was not 
supported by a large number of interviewees. The 
decision to design collaboration in a specific course 
should be based upon several factors, including the 
difficulty of the content, course level, subject matter, 
and maturity of the students. 

►M Courses are separated into self-contained segments 
(modules) that can be used to assess student mastery 
before moving forward in the course or program (#18). 

►M The modules I segments are of varying lengths determined 
by the complexity of learning outcomes (#1^)- 

Similar to the arguments about collaboration, many 
faculty and administrators did not view modular 
learning as a requirement for quality. A few faculty, 
however, insisted that modular learning is a vital 
pedagogical activity. Others were equally insistent that 
there are several reasons for not designing a course or 
other learning activities in self-contained segments, as 
knowledge was integrated across several courses. The 
weight of evidence suggested that requiring modular 
learning as a prerequisite for quality distance education 
is inappropriate. 

Course Structure Benchmarks 

►M Specific expectations are set for students with respect to 
a minimum amount of time per week for study and 
homework assignments (#25). 

►M Faculty are required to grade and return all assignments 
within a certain time period (#26). 

Many faculty and administrators were concerned about 
the prescriptive nature of these benchmarks regarding 
time expectations for students and faculty. Internet-based 
distance education is complex, and the pedagogy is 
constantly evolving as faculty experiment with a variety 
of techniques. Moreover, much of the learning process is 
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self-paced/ which argues against the imposition of rigid 
standards. However, there was strong agreement that 
students and instructors each must have clear expectations 
of their responsibilities which should be no different than 
for any course, irrespective of its delivery. 

New Benchmarks 

There were three quality benchmarks that were not 
included in the original 45 but were recommended by 
a large majority of faculty and administrators and are 
consistent with the outlined criteria. 

►H The reliability of the technology delivery system is as 
failsafe as possible. 

At every one of the institutions in this case study, a 
benchmark related to the importance of the reliability 
of technology was suggested. As stated by many, system 
crashes were unacceptable and must be avoided at all 
costs. When the system is down, essentially little to no 
learning takes place. If the system is down for any length 
of time, students often become discouraged, angry, and 
begin to lose interest. Even if the system is down for a 
short period of time, the experience is disruptive to 
students and fosters skepticism that it will happen again. 
Several faculty and administrators noted that student 
attrition is often related to system unreliability. 

►W Faculty and students agree upon expectations 
regarding times for student assignment completion 
and faculty response. 

As noted earlier, there are at least two benchmarks 
regarding faculty response to student assignments and 
feedback. There was strong agreement that faculty 
should respond to students as quickly as possible and 
that students need to spend a sufficient amount of time 
online to achieve academic success. The benchmarks 
were, however, too directive to be of much value. Many 
faculty and administrators recognized the potential of 
the Internet to transform the teaching/leaming process 
and any benchmarks that inhibited their ability to 
innovate and experiment were rejected. This 
benchmark reinforces interaction and constructive 
faculty and student relationships without constraining 
pedagogical innovation. 

►W Questions directed to student service personnel are 
answered accurately and quickly. 



Because online students may never have the opportunity 
to visit the campus from which they are taking courses 
or receiving their degree, it is essential that they enjoy 
the same access as traditional students to services such 
as admissions, registration, financial aid, career 
counseling, and academic course requirements. In fact, 
because they often do not have the opportunity for any 
face-to-face communication with student service 
personnel, extraordinary methods need to be developed 
to ensure that online students do not feel abandoned. 
These methods were discussed earlier in the report. 

Benchmarks That Are Essential for Quality 
Internet-based Distance Education 

The following represents the final list of benchmarks 
resulting from this study. These benchmarks are the 
most essential to the success of an Internet-based 
distance education program at any institution. These 
benchmarks may be useful to government 
policymakers, institutional decisionmakers, faculty, 
and students, as well as others with an interest in 
ensuring that the highest quality of higher education 
possible is being provided via Internet-based programs. 

Institutional Support Benchmarks 

►H A documented technology plan that includes 
electronic security measures (i.e., password 
protection, encryption, back-up systems) is in place 
and operational to ensure both quality standards 
and the integrity and validity of information. 

►H The reliability of the technology delivery system is 
as failsafe as possible. 

►H A centralized system provides support for building 
and maintaining the distance education 
infrastructure. 

Course Development Benchmarks 

►H Guidelines regarding minimum standards are used 
for course development, design, and delivery, while 
learning outcomes — not the availability of existing 
technology — determine the technology being used 
to deliver course content. 

►M Instructional materials are reviewed periodically 
to ensure they meet program standards. 
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►W Courses are designed to require students to engage 
themselves in analysis, synthesis, and evaluation 
as part of their course and program requirements. 

Teaching/Leaming Benchmarks 

►M Student interaction with faculty and other 
students is an essential characteristic and is 
facilitated through a variety of ways, including 
voice-mail and/or e-mail. 

►M Feedback to student assignments and questions is 
constructive and provided in a timely manner. 

►M Students are instructed in the proper methods of 
effective research, including assessment of the 
validity of resources. 

Course Structure Benchmarks 

►W Before starting an online program, students are 
advised about the program to determine ( 1 ) if they 
possess the self-motivation and commitment to 
learn at a distance and (2) if they have access to the 
minimal technology required by the course design. 

►M Students are provided with supplemental course 
information that outlines course objectives, 
concepts, and ideas, and learning outcomes for 
each course are summarized in a clearly written, 
straightforward statement. 

►M Students have access to sufficient library resources 
that may include a "virtual library" accessible 
through the World Wide Web. 

►M Faculty and students agree upon expectations 
regarding times for student assignment completion 
and faculty response. 

Student Support Benchmarks 

►M Students receive information about programs, 
including admission requirements, tuition and 
fees, books and supplies, technical and proctoring 
requirements, and student support services. 



►H Students are provided with hands-on training and 
information to aid them in securing material through 
electronic databases, interlibrary loans, government 
archives, news services, and other sources. 

►H Throughout the duration of the course/program, 
students have access to technical assistance, 
including detailed instructions regarding the 
electronic media used, practice sessions prior to 
the beginning of the course, and convenient access 
to technical support staff. 

►M Questions directed to student service personnel are 
answered accurately and quickly, with a structured 
system in place to address student complaints. 

Faculty Support Benchmarks 

►W Technical assistance in course development is 
available to faculty, who are encouraged to use it. 

►W Faculty members are assisted in the transition from 
classroom teaching to online instruction and are 
assessed during the process. 

►H Instruaor training and assistance, including peer 
mentoring, continues through the progression of 
the online course. 

►M Faculty members are provided with written 
resources to deal with issues arising from student 
use of electronically-accessed data. 

Evaluation and Assessment Benchmarks 

►M The program's educational effectiveness and 
teaching/learning process is assessed through an 
evaluation process that uses several methods and 
applies specific standards. 

►H Data on enrollment, costs, and successful/ 
innovative uses of technology are used to evaluate 
program effectiveness. 

►H Intended learning outcomes are reviewed regularly 
to ensure clarity, utility, and appropriateness. 
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Appendix One: Institutional Profiles 

Brevard Community Collese Cocoa/ FL 

Carnegie Class: Associate of Arts Enrollment: 14,732 Established: 1960 

Brevard Community College began offering distance education telecourses in 1974 and has made distance learning 
via the Internet a priority in recent years. Currently, two entire associate's degree programs can be completed either 
online or through telecourses, as well as a number of individual courses. Within the next three to five years the 
institution plans to expand the number of programs that can be completed at a distance, focusing on increasing 
student retention in distance classes. Brevard also is a participant in the PBS Adult Learning Service "Going the 
Distance Program," which brings higher education to the growing numbers of adult students interested in distance 
education. The institution has a comprehensive plan to improve the quality of teaching and productivity of students 
through the use of technology and distance education. Brevard will be assisted in achieving these goals through a 
grant from the U.S. Department of Education's Fund for the Improvement of Postsecondary Education (FIPSE) that 
is focused on faculty training and development in order to impact positively student learning. 



Regents College Albany, NY 

Carnegie Class: Baccalaureate II Enrollment: 17,358 Established: 1970 

Regents originally began as the External Degree Program of the University of the State of New York and operates on 
the philosophy that "What you know is more important than where or how you learned it." In April 1998, the 
college was granted an independent charter by the New York Board of Regents and now operates as a private institution. 
Focused on working adults, Regents provides an independent study program for various degrees ranging from Nursing 
to Information Technology. Students work at their own pace and can take examinations to demonstrate their 
knowledge. This format has allowed over 83,000 people worldwide to earn a college degree through Regents' programs. 
In February 1999 Regents was awarded a Meritorious Course Award from the University Continuing Education 
Association in recognition of its Theoretical Frameworks of Nursing Practice course. 



University of Illinois at Urbana-Champaign Urbana, IL 

Carnegie Class: Research I Enrollment: 36,019 Established: 1867 

The University of Illinois at Urbana-Champaign is very active in distance learning on a number of different fronts. 
They are one of three participants in the University of Illinois Online (along with the Springfield and Chicago 
campuses), which offers 20 degree- or certificate-programs over the Internet with more in development. The university 
also offers online master's degree programs in six fields. The school received funding from the Pew Grant Program 
of Course Redesign in 1999 to redesign a two-course sequence in Economic Statistics to make the courses more 
"hands on." Another innovative facet of the university's distance education programs is the Idea Village. Still in its 
initial phases, the Idea Village is essentially an online college community that will feature portals to online classrooms, 
academic and social information, an online library, and an online help center. 
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University of Maryland University College College Park, MD 

Carnegie Class: Master's I Enrollment: 13,786 Established: 1947 

A virtual institution, the University of Maryland University College (UMUC) was founded with the mission of 
providing continuing education to Maryland's professional workforce. With experience in distance education that 
spans more than a quarter century UMUC began offering courses over the Internet in Fall 1997 and currently offers 
14 bachelor's and 10 master's degree programs online. The institution operates internationally, serving active duty 
military personnel in Europe, the Mid-East, the Pacific, and East Asia. To assure that increasing numbers of students 
are well served, UMUC is focusing on infrastructure issues, including expansion of student services over the World 
Wide Web as well as the availability of library resources online. UMUC was awarded the University Continuing 
Education Association's Award for Innovative Distance Education in 1998 and 1999, and was included in Forbes 
Magazines list of the top 20 "cyber universities" in 1997. 

Utah State University Logan, UT 

Carnegie Class: Research I Enrollment: 21,234 Established: 1888 

Utah State University (USU) has been involved in various forms of distance education — from correspondence courses 
to satellite-delivered courses — since 1911. The institution has now expanded those offerings to include two types of 
online courses, semester-based and independent study. The semester-based courses are similar to the courses that 
students would take on campus and follow a similar schedule. The independent study courses allow students to 
enroll at anytime with the provision that they must complete the course within one calendar year of enrolling. 
Additionally, USU is now offering a number of degree programs online, mostly from the Department of Special 
Education and Rehabilitation. USU also has been selected as a participant in the Learning Anytime Anywhere 
Partnerships program created in the 1998 reauthorization of the Higher Education Act. It will partner with PBS 
Adult Learning Service, the American Association for Higher Education, Western Governors University, and MadDuck 
Technologies to find innovative ways of providing persons with disabilities access to postsecondary education. 



Weber State University Ogden, UT 

Carnegie Class: Master's II Enrollment: 14,613 Established: 1889 

Weber State University (WSU) is heavily invested in distance learning. The institution began offering independent 
study courses over 10 years ago, and in Fall 1995 conceptualized an online campus. The plan included moving 
courses to the Internet, and providing student services, academic advising, and technical support. Over the next two 
years the institution designed and tested all of its own tools including online chat and bookstore services. In Fall 
1997, WSU Online was launched. Currently it offers two-thirds of the online courses in Utah. These include over 70 
independent study courses offered in more than 20 disciplines. WSU also offers a number of associate's and bachelor's 
degree programs in health professions. Despite the fact that the online venture is only two years old, it now enrolls 
more than 2000 students and recieved an Innovation in Distance Education Award from the University Continuing 
Education Association in 1998. 

Carnegie classification, enrollment, and establishment data are taken from the 1999 Higher Education Directory. 
Higher Education Publications, Falls Church, VA. Enrollment data are from 1997-98. 
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Institutional Support 




1. Faculty are provided professional incentives for innovative practices 


4.1 


1.943 


83 
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to encourage development of distance learning courses. 
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2. There are institutional rewards for the effective 
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teaching of distance learning courses. 
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3. A documented technology plan is in place to ensure quality standards. 
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1.796 
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4. Electronic security measures are in place to ensure 
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1.183 
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the integrity and validity of information. 
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5. Support for building and maintaining the distance 
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1.606 
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education infrastructure is addressed by a centralized system. 
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Course Development 










6. Distance learning course development must 
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be approved through a broad peer review process. 
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7. Guidelines exist regarding minimum standards 
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1.789 
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for course development, design, and delivery. 
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8. Course design is managed by teams comprised of faculty, content 


4.4 


2.083 
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experts, instructional designers, technical experts, and evaluation personnel. 
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9. During course development, the various learning styles 


4.6 


1.829 
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of students are considered. 












- “ ■■■ • ■ - ■ — 


'■"4:2 




z”z?7' z:: 


5 


10. Assessment instruments are used to ascertain the specific learning 


2.8 


1.737 
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styles of students, which then determine the type of course delivery. 
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11. Courses are designed with a consistent structure. 


5.2 


1.692 
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easily discemable to students of varying learning styles. 
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12. The technology being used to deliver course 
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1.736 
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content is based on learning outcomes. 
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13. Instructional materials are reviewed periodically 
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1.723 


87 


7 


to ensure they meet program standards. 
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Teachin^/Learnins Process 










14. Student interaction with faculty is facilitated through a variety of ways. 
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15. Student interaction with other students 
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is facilitated through a variety of ways. 
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16. Feedback to student assignments and 


5.6 


1.273 
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questions is provided in a timely manner. 
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Note: The set of values presented represent the presence and importance of each benchmark. 
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All Survey Results 
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17. Feedback to students is provided in a manner 
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1.219 
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that is constructive and non-threatening. 
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18. Courses are separated into self-contained segments (modules) 


5.4 


1.814 


117 


7 


that can be used to assess student mastery before moving 










forward in the course or program. 
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19. The modules/segments are of varying lengths 
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1.780 


71 


7 


determined by the complexity of learning outcomes. 
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20. Each module/segment requires students to engage themselves in 
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analysis, synthesis, and evaluation as part of their course assignments. 
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21. Class VO ice- mail and/or e-mail systems are provided to encourage 
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1.570 
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students to work with each other and their instructor(s). 
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22. Course are designed to require students to work in groups utilizing 
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problem-solving activities in order to develop topic understanding. 
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23. Course materials promote collaboration among students. 
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Course Structure 










24. Students are provided with supplemental course information 


6.1 


1.125 
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that outlines course objectives, concepts, and ideas. 
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25. Specific expectations are set for students with respect to a minimum 
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amount of time per week for study and homework assignments. 














71.4:211::,. 


77:71:0417^ 


77418 7 7 


I7...577Z7 


26. Faculty are required to grade and return all 
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assignments within a certain time period. 
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27. Sufficient library resources are made available to the students. 
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28. Students are instructed in the proper methods of effective 
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1.827 


116 


7 


research, including assessment of resource validity. 
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29. Before starting the program, students are advised 
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about the program to determine if they have the 










self-motivation and commitment to learn at a distance. 
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30. Learning outcomes for each course are summarized 
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in a clearly written, straightforward statement. 
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Student Support 










31. Students can obtain assistance to help them 
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use electronically accessed data successfully. 
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32. Students are provided with hands-on training and information 
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1.834 


126 
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to aid them in securing material through electronic databases. 










interlibrary loans, government archives, news services, etc. 
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33. Written information is supplied to the student about the program. 
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34. Easily accessible technical assistance is available to all 


5.4 


1.743 
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students throughout the duration of the course/program. 
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35. A structured system is in place to address student complaints. 
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Faculty Support 










36. Technical assistance in course development is 
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available to faculty and they are encouraged to use it. 
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37. Faculty members are assisted in the transition from classroom 
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teaching to distance instruction and are assessed in the process. 










^ ' ” , ; ; 1. 1’’ 1 717 


77:4:i777~ 


77 a7^ 7 


77 :?:577_ ; 


_7 717777 


38. There are peer mentoring resources available to 
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faculty members teaching distance courses. 
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39. Distance instructor training continues throughout 
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1.764 
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the progression of the online class. 


: ...1437177 


"0,773 


: :zMi.z z 


:.z:i::...:7:z 


40. Faculty members are provided with written resources to deal 
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with issues arising from student use of electronically-accessed data. 
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Evaluation and Assessment 










41. The program s educational effectiveness is measured using several methods. 
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42, An evaluation process is used to improve the teaching/leaming process. 
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43. Specific standards are in place to compare and improve learning outcomes. 
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44. Data on enrollment, costs, and successful/innovative uses 
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67 


5 


of technology are used to evaluate program effectiveness. 
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45. Intended learning outcomes are regularly reviewed 
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to ensure clarity, utility, and appropriateness. 
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